Stochastic Dropout: Activation-level Dropout to Learn Better Neural Language Models

Recurrent Neural Networks are very powerful computational tools that are capable of learning many tasks across different domains. However, it is prone to overfitting and can be very difficult to regularize. Inspired by Recurrent Dropout [1] and Skip-connections [2], we describe a new and simple regularization scheme: Stochastic Dropout. It resembles the structure of recurrent dropout, but offers skip-connection over the recurrent depth. We reason the theoretical construct of such method and compare its regularization effectiveness with feedforward dropout and recurrent dropout. We demonstrate that Stochastic Dropout not only offers improvement when applied to vanilla RNN models, but also outperforms feedforward Dropout on word-level language modeling. At last, we show that the model can achieve even better result if stochastic dropout and feedforward dropout are combined.